
Calhoun 

iniQiuiic^iul Ar{hiv« of tilt Mil vdl Poii^roduiit School 


Calhoun: The NPS Institutional Archive 
□Space Repository 



Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


2019-12 

RELIABILITY GROWTH MODELING OF A 
COMPLEX SYSTEM 

Davis, Carol B. 

Monterey, CA; Naval Postgraduate School 


http://hdl.handle.net/10945/64131 


This publication is a work of the U.S. Government as defined in Title 17, United 
States Code, Section 101. Copyright protection is not available for this work in the 
United States. 

Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


htt p://w ww. n ps. e du/l ib ra ry 


Caflwuo is the Naval Postgraduate School's public access digital repository for 
research mate rials and institutiional publicatkins created by the NPS community. 
Calhoun is named for Professor of Mathematics Guy K. Caftiouo, NPS's first 
appointed — and published — schoteily author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 University Circle 
Monterey, California USA 93943 







NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY, CALIFORNIA 


THESIS 


RELIABILITY GROWTH MODELING OF A COMPLEX 

SYSTEM 


by 


Carol B. Davis 


December 2019 


Thesis Advisor: 

Bryan M. O’Halloran 

Co-Advisor: 

John M. Green 


Approved for public release. Distribution is unlimited. 




THIS PAGE INTENTIONALLY LEET BLANK 



REPORT DOCUMENTATION PAGE 

Form Approved OMB 

No. 0704-0188 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing 
instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of 
information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions 
for reducing this burden, to Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson 
Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project 
(0704-0188) Washington, DC 20503. 

1. AGENCY USE ONLY 2. REPORT DATE 

(Leave blank) December 2019 

3. REPORT TYPE AND DATES COVERED 

Master’s thesis 

4. TITLE AND SUBTITLE 

RELIABILITY GROWTH MODELING OF A COMPLEX SYSTEM 

5. LENDING NUMBERS 

6. AUTHOR(S) Carol B. Davis 

7. PEREORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Naval Postgraduate School 

Monterey, CA 93943-5000 

8. PEREORMING 
ORGANIZATION REPORT 
NUMBER 

9. SPONSORING / MONITORING AGENCY NAME(S) AND 

ADDRESS(ES) 

N/A 

10. SPONSORING / 
MONITORING AGENCY 
REPORT NUMBER 

11. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the 
official policy or position of the Department of Defense or the U.S. Government. 

12a. DISTRIBUTION / AVAILABILITY STATEMENT 

Approved for public release. Distribution is unlimited. 

12b. DISTRIBUTION CODE 

A 

13. ABSTRACT (maximum 200 words) 

Acquisition Category (ACAT) Level I programs are held to a mandated reliability growth requirement 
by the Milestone Decision Authority (MDA). Many times, ACAT Level I programs do not have the data 
required prior to going to test in order to assess and determine the requirement posed on them by the MDA. 
Most times, due to the complexity of the ACAT Level I program, there is not a comparable program in 
which to use its data. While there are existing methods that assess reliability growth, they require having 
failure data available. Due to the complexity of ACAT Level I programs, having the type of data available 
early in the program’s schedule is usually not possible. Therefore, the existing methods do not work to 
answer the mandated requirement posed on such complex programs. To overcome this challenge, a design 
methodology was developed to exhibit the results of reliability growth for ACAT Level I programs. This 
thesis (1) prepares data gathered from the program’s documents as an input to the AMSAA PM2-C model to 
develop a baselined reliability growth planning curve, (2) produces a demonstrated reliability growth curve, 
and then (3) compares the baselined planning curve to the demonstrated reliability growth curve. These tasks 
are captured within the design methodology, which is presented as a repeatable process applicable to ACAT 
Level I programs. Executing the series of steps in this thesis fulfills the mandated requirement posed by the 
MDA with fidelity. 

14. SUBJECT TERMS 

reliability, growth curves, growth modeling, reliability modeling, reliability growth, 
modeling reliability growth curves 

15. NUMBER OE 
PAGES 

73 

16. PRICE CODE 

17. SECURITY 
CLASSIEICATION OE 
REPORT 

Unclassified 

18. SECURITY 
CLASSIEICATION OE THIS 
PAGE 

Unclassified 

19. SECURITY 
CLASSIEICATION OE 
ABSTRACT 

Unclassified 

20. LIMITATION OE 
ABSTRACT 

UU 


NSN 7540-01-280-5500 


1 


Standard Form 298 (Rev. 2-89) 
Prescribed by ANSI Std. 239-18 




























THIS PAGE INTENTIONALLY LEET BLANK 


11 



Approved for public release. Distribution is unlimited. 


RELIABILITY GROWTH MODELING OF A COMPLEX SYSTEM 


Carol B. Davis 

Civilian, Department of the Navy 
BS, University of North Carolina at Charlotte, 2011 
MS, Old Dominion University, 2015 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN SYSTEMS ENGINEERING MANAGEMENT 

from the 

NAVAL POSTGRADUATE SCHOOL 
December 2019 


Approved by: Bryan M. O’Halloran 
Advisor 


John M. Green 
Co-Advisor 


Ronald E. Giaehetti 

Chair, Department of Systems Engineering 



THIS PAGE INTENTIONALLY LEET BLANK 


IV 



ABSTRACT 


Acquisition Category (ACAT) Level I programs are held to a mandated reliability 
growth requirement by the Milestone Deeision Authority (MDA). Many times, ACAT 
Level I programs do not have the data required prior to going to test in order to assess and 
determine the requirement posed on them by the MDA. Most times, due to the 
eomplexity of the ACAT Level I program, there is not a eomparable program in whieh to 
use its data. While there are existing methods that assess reliability growth, they require 
having failure data available. Due to the complexity of ACAT Level I programs, having 
the type of data available early in the program’s sehedule is usually not possible. 
Therefore, the existing methods do not work to answer the mandated requirement posed 
on sueh complex programs. To overeome this challenge, a design methodology was 
developed to exhibit the results of reliability growth for ACAT Level I programs. This 
thesis (1) prepares data gathered from the program’s doeuments as an input to the 
AMSAA PM2-C model to develop a baselined reliability growth planning curve, (2) 
produees a demonstrated reliability growth eurve, and then (3) eompares the baselined 
planning curve to the demonstrated reliability growth eurve. These tasks are eaptured 
within the design methodology, whieh is presented as a repeatable proeess applieable to 
ACAT Level I programs. Exeeuting the series of steps in this thesis fulfills the mandated 
requirement posed by the MDA with fidelity. 
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EXECUTIVE SUMMARY 


Acquisition Category (ACAT) Level I programs are held to a mandated reliability 
growth requirement by the Milestone Deeision Authority (MDA). At the ACAT Level 
designation of one (I), the MDA has a speeial interest and imposes additional mandated 
requirements to produce a system that meets the warfighters emergent need. Reliability 
growth modeling is one of the mandated requirements imposed by the MDA. It is due to 
the mandated requirement (i.e., reliability growth modeling) that the need exists in 
developing a design methodology to fulfill the requirement and meet the MDA’s approval. 

Many times, ACAT Level I programs do not eurrently have the data prior to testing 
that is required in order to verify the reliability requirement. There are many reliability 
methodologies that exist (i.e., the Duane model, the U.S. Army Material Systems Analysis 
Aetivity (AMSAA) Growth Traeking model, AMSAA’s Crow Projeetion model, and 
AMSAA’s Maturity Projeetion model). It is important to note that these existing methods 
require failure data. Due to the eomplexity of ACAT Level I programs, having the type of 
data available early in the program’s sehedule is usually not possible. Therefore, existing 
methods do not work to answer the mandatory requirement posed on such complex 
programs. 

Given this limitation, this thesis foeuses on the development of a design 
methodology to address and answer the need that exists. This design methodology uses the 
AMSAA’s Projeetion Methodology 2-Continuous (PM2-C) model for eomplex ACAT 
Level I systems through an iterative proeess. This thesis (1) prepares data gathered from 
the program’s doeuments as an input to the AMSAA PM2-C model to develop a baselined 
reliability growth planning curve, (2) produces a demonstrated reliability growth curve 
based on system test data, then (3) eompares the baselined planning eurve to the 
demonstrated reliability growth eurve to understand the if the requirement has been met. 
These tasks are eneapsulated within design methodology, whieh is presented as a 
repeatable proeess applieable to ACAT Level 1 programs. 
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The purpose of this design methodology is to allow complex systems the structure 
and the steps to predict planned reliability growth in comparison to achieved reliability 
results (i.e., demonstrated curve). This design methodology is demonstrated with a case 
study. The case study looks at the Phoenix Talon Program, and its execution of the design 
methodology that assesses and reports reliability growth. Executing this design 
methodology produces an iterative reliability growth testing process to be assessed at the 
conclusion of each test phase in order to ensure the system is producing increased (or 
stable) reliability throughout the milestone and not just assessed at a single instance (once) 
at the conclusion. 

Should the design methodology in this research not be utilized, ACAT Level I 
programs may continue to suffer in changing the baselined planning reliability growth 
curve at the conclusion of all test phases. ACAT Level I programs across the Department 
of Defense would benefit from using the design methodology in this thesis, which 
systematically tests the initial baselined planning curve to assess the achievements in 
reliability growth to the unchanging baseline; and to do so iteratively in order to mitigate 
any declines in reliability growth amongst test phases. 
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I. INTRODUCTION 


An acquisition program is a directly funded effort designed to provide an improved 
eapability in response to a valid operational need. An Aequisition Category (ACAT) level 
designation of one (I) is when all inerements of the program surpasses a particular dollar 
value. The dollar value is estimated to be more than $2.79 billion dollars (USD AT&L 
2014, 3). At this level, the Milestone Deeision Authority (MDA) has a speeial interest in 
eomplex programs of sueh dollar value, and imposes additional mandated requirements to 
ensure the federal governments dollars are spent to produee a system that meets the 
warfighters’ emergent need. Reliability growth modeling is one of the mandated 
requirements imposed by the MDA. It is due to this mandated requirement (i.e., reliability 
growth modeling) that the need exists in developing a design methodology to fulfill the 
requirement and meet the MDA’s approval. Should a program of ACAT Level I 
designation not meet these mandated requirements to the MDA, they will not be rewarded 
to move to the next milestone in the aequisition proeess. As such, this thesis develops a 
design methodology speeifically to address this reliability growth modeling requirement. 

While models used in engineering help visualize systems, predieting is not an exaet 
seienee. Model developers, systems engineers, and analysts need a medium that will 
aeeurately and consistently illustrate growth of their ACAT Level I system based on 
planned to and demonstrated test data. However, an approaeh like this to model reliability 
growth eurrently does not exist. While there are multiple methods for modeling reliability 
growth, none exist for establishing a baselined reliability growth eurve and for testing it 
against aehieved results from test data. The next ehapter will address the history of 
reliability growth modeling as well as state partieular methods used. However, these 
methods are not sufficient for ACAT Level I programs due to the lack of data for 
comparison at the time of meeting with the MDA to be granted testing approval. 

A, RESEARCH OBJECTIVE AND METHODOLOGY 

To address the ehallenges presented, the research objective provides three 
eontributions ineluding (1) prepares data gathered from the program’s doeuments as an 
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input to the AMSAA PM2-C model to develop a baselined reliability growth planning 
curve, (2) produces a demonstrated reliability growth curve based on system test data, then 
(3) compares the baselined planning curve to the demonstrated reliability growth curve to 
understand the if the requirement has been met. These tasks are encapsulated within the 
design methodology, which is presented as a repeatable process applicable to ACAT Level 
1 programs. Executing the series of steps following the research methodology fulfills the 
requirement posed by the MDA with fidelity. 

The research methodology begins by glancing into the history of reliability growth 
to give a background as to its importance, as well as applicable related works of reliability 
in Chapter II. Due to the requirements of the AMSAA PM2-C model, this information is 
diverse and requires specific consideration (see Chapter III). With the availability of a data 
set, the AMSAA PM2-C model is then used to predict a reliability growth curve. Chapter 
IV reduces ambiguity for specific elements of the AMSAA PM2-C model to include the 
program’s reliability requirement and what is meant by that, as well as the program’s 
degradation factor and implied threshold values. The third contribution, the development 
of a design methodology (see Chapter V), wraps the first two contributions, as well as 
demonstrated reliability, into a practical process useful specifically for ACAT Level 1 
programs. This method is intended to show the MDA that throughout the milestone several 
instances of reliability testing was conducted and growth was achieved (demonstrated) 
across the system test phase. Having such a design methodology offers the practitioner a 
repeatable process to implement the reliability growth modeling and system test results. 
The methodology also demonstrates the trend of demonstrated reliability growth relative 
to the requirement, which clearly identifies when the system’s reliability growth is not 
sufficiently improving and is likely to not achieve the desirable reliability requirement. In 
Chapter VI, this thesis is illustrated in a case study (Phoenix Talon) where it (1) prepares 
data gathered from the Phoenix Talon’s program documents as an input to the AMSAA 
PM2-C model to develop a baselined reliability growth planning curve, (2) produces a 
demonstrated reliability growth curve based on system test data, then (3) compares the 
baselined planning curve to the demonstrated reliability growth curve to understand the if 
the requirement has been met. These tasks are encapsulated within the design methodology. 
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which is presented as a repeatable process applieable to AC AT Level 1 programs. Figure 
1 Methodology Flowehart highlights the methodology of this research in a flowchart. The 
following chapters address each of the steps in the flowchart. 



Figure 1. Methodology Flowchart 


B. RESEARCH QUESTION TO BE SOLVED 

How do ACAT Level I programs demonstrate reliability growth modeling to meet 
the mandated requirement imposed by the MDA? The benefit of this research and thesis is 
to accurately and consistently answer the DoDI 5000.02 requirement (USD AT&L 2014, 
3-49) for reliability growth for ACAT Level I programs. As it states in the DoDI 5000.02, 
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Reliability Growth Curves (RGCs) will refleet the reliability growth 
strategy and be employed to plan, illustrate, and report reliability growth. 

RGCs are included in the TEMP and submitted at the conclusion of 
Milestone B. RGCs are in a series of intermediate goals and tracked through 
fully integrated, system-level test and evaluation events at least until the 
reliability threshold is achieved. If a single curve is not adequate to describe 
overall system reliability, curves for critical subsystems should also be 
employed. (USD AT&L 2014, 3) 

This design methodology compares the U.S. Army Material Systems Analysis 
Activity (AMSAA) Projection Methodology 2-Continuous (PM2-C) model through an 
iterative process and the best approach to take when scheduling phases into the 
Developmental Test (DT) plan. As a result, this can allow for programs to better plan their 
test phases and hours to ensure that the identified baseline correlates with realistic 
expectations of reliability growth. This design methodology allows for demonstrating 
reliability growth from baseline to completion of the acquisition milestone. 
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II. LITERATURE REVIEW 


This chapter provides a baekground to reliability growth modeling. This 
background focuses on discussing the history of reliability growth as well as providing a 
literature review in order to eapture the related works as it direetly applies to reliability 
growth modeling. The background a brief synopsis into how reliability growth eame into 
existenee as well as the researeh efforts that have addressed reliability growth unique to 
their system. However, the related works are not suffieient for addressing the need that 
exists for the mandated reliability growth modeling speeifie to ACAT Level I programs. 

A. HISTORY OF RELIABILITY GROWTH MODELING 

This subseetion diseusses the history of reliability growth modeling and establishes 
an understanding of how the models have progressed. In eontrast, the next subseetion 
diseusses reliability researeh. 

Reliability growth methodologies have been in use sinee introdueed in 1964 (Meth 
1992). While it is likely that engineers and statistieians have been using reliability 
measurements to determine growth prior to 1964, it is this year that Duane had his first 
highly influential paper, whieh led to reliability growth measuring and methodologies 
beeoming broadeasted. The broadeasted methodologies led to influential applieations of 
reliability growth sueh as Crow, Jewell, and Ushakov (National Researeh Couneil 2002). 
Sinee its introduetion, reliability growth eoncepts and applieations have provided the basis 
for assessing newly developed aequisition programs and determining whether they are 
reliable enough to meet the warfighters needs or not (National Researeh Council 2002). 

The Duane model was the popular methodology between 1964 and into the 1970s. 
Program Managers (PMs) were able to predict the future of the systems reliability based 
on its fit to the Duane curve that the model ealeulated. Even though the Duane model was 
able to predict reliability, it did not give PMs and other stakeholders of the program any 
understanding about the growth proeess itself Many were asking, “If we are only here, 
how do we get there?” (Duane 1964, 566). The immediate response and reaetion from the 
question was that improvements would lead to inereased reliability, as long as something 
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improved the system. Again, the questions started rising, “but now are we making the right 
reliability improvements to our system?” (566). 

In 1984, Crow, while working at the U.S. Army Materiel System Analysis Aetivity 
(AMSAA), further developed the Duane model in order to make it more praetieal for 
programs. Crow attempted to answer the questions that arose after years of using the Duane 
eurve to prediet and assess reliability growth. Crow’s modifieations to the Duane model 
allowed greater flexibility. Whereas the Duane model applied data loeally. Crow’s model 
helped to refine reliability growth and predietion modeling for longer periods, extending 
out to the life eycle of the system itself. 

However, in building upon the Duane model and later the “Refined Duane model” 
(the Crow model), questions remained as to “how do we predict the reliability growth of 
the system for planning purposes?” and “how do we predict reliability growth when no test 
data exists yet?” (National Research Council 2002, 10). Until now, programs were able to 
use data from a comparable system to meet this requirement. However, as the complexity 
of AC AT Level I programs have increased and now encompass several systems into one, 
utilizing data from a single comparable system will no longer suffice. 

B, RELATED RESEARCH FOR RELIABILITY GROWTH MODELING 

Over the past several years, studies have demonstrated reliability growth for various 
programs. While the body of related work is relatable to this thesis, it does leave out the 
fundamental requirement of demonstrating and comparing the baselined reliability growth 
curve to achieved reliability results in a demonstrated curve throughout the systems 
acquisition milestone. Today’s DoDI requirement for AC AT Level I entails that a plan 
must be in place prior to granted funding to demonstrate and test the conceptual design of 
the system. This subsection highlights the comparable studies as well as a reflection of the 
literature with its application to this thesis. 

In 2005, Johnson used a “hierarchical model for estimating the reliability of 
complex systems” (Johnson 2005, 224). He discussed the use of a Bayesian model “to 
assess early reliability of complex systems, for which no failure data on the system was 
currently available” (Johnson 2005, 225). He used failure data used from a comparable 
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system in order to determine the new systems early reliability. The case study in this thesis 
does not have component failure data from a comparable system because a comparable 
system does not exist. Most ACAT Level I systems do not have comparable systems in 
which to reveal higher-level similar component failure data from due to the systems 
complexity. 

In 2011, Hsu’s research pertained to an adaptive reliability analysis using path 
testing for complex component-based software systems (Hsu 2011, 158-170). His paper 
discussed the use of an adaptive framework for incorporating path testing into estimating 
the reliability for modular software systems. Hsu’s reliability testing methodology 
highlighted software in loop structures. Hsu proposed methods and paths assisted in 
estimating software reliability for in the early stages of software development. Although 
his research does closely compare to my thesis in the fact that ACAT Level I programs are 
complex and usually very software intense, it focuses on complex systems using reliability 
modeling to assess the aspect of software reliability with software development, making 
his methods only applicable to software. Hsu’s proposed methods will not be the focus of 
my thesis. This thesis utilizes testing planning data and test results in order to model the 
complex system’s reliability, which encompasses a hardware system that is software 
intense. In addition, his paper assisted in creating a framework for software with loop 
structures. This thesis does not discuss looping structures, although it is very common that 
loops and timing mechanisms are in-place in software intense complex systems. 

Similarly, in 2006 Bedford wrote a dissertation on expert elicitation for reliable 
system design, which also aligns with this thesis’s concern for reliability growth (Bedford 
2006, 428-450). His dissertation focused on the development of failure mitigation 
strategies to account for reliability potential for the life cycle of the system. Bedford shows 
predictive reliability from data of a comparable system and assesses the systems strategies 
for future assessments. This thesis discusses such similarities but does not have data from 
a comparable system due to the complexity of ACAT Level I programs. Bedford’s 
dissertation also does not discuss the use of ACAT Level I programs and the requirements 
mandated for them in reference to reliability modeling. 


7 



Gokhale (1998) briefed researeh findings similar to Bedford’s that diseussed the 
reliability simulation of eomponent-based software systems and foeused on the life eyele 
“end of life” style approaeh (Gokhale 1998, 111-123). Gokhale’s research demonstrated 
how discrete event simulations portrayed the reliability of complex systems at the 
individual component level. Gokhale’s approach allowed investigations into the failures 
shown in an instantaneous manner and resulted in repairing the failure and continuing 
operations based outcome. His real-environment application influenced the simulations 
executed and resulted in anticipated reliability growth modeling. One approach to 
measuring the reliability growth of a system may be to break the system down into its lower 
component levels, each with its own reliability and failure data. In my thesis, test phase 
data is acquired as the design methodology is applied at the system level from the achieved 
results described in Chapter VI, Case Study Demonstration, which more accurately 
represents the system, as it does not solely rely on the reliability data from its manufacturer. 
This is because the data represents the system explicitly whereas data from manufacturers 
and other systems come from environments, operational uses, and other reliability stressors 
that are either unknown or not applicable. 

In 2001, Weckman developed research that highlighted modeling reliability of 
repairable systems in the aviation industry (Weckman 2001, 51-63). Weckman’s research 
discussed reliability as it pertains to looking into a jet engine. The jet engine requires repair 
and restoration after a certain amount of time and periodically throughout its life cycle. His 
paper suggests using the Weibull process as an additional tool to assess the reliability of 
repairable systems. The jet engine acts as an example of demonstrating model predictions 
and predicting reliability. Weckman’s approach is very similar theoretically to this thesis. 
However, instead of a simple Weibull model that looks into modeling the component (i.e., 
the jet engine), it assesses the system as a whole. In addition, this thesis accounts for 
software failures (quickly repairable) as well as hardware failures (repairable but time- 
consuming) and allows the model to be tailored to assess failure modes (both instantly 
repaired and those that can be repaired over time (i.e., predicted)). The AMSAA PM2-C 
model utilizes the Poisson process whereas Weckman utilizes the Weibull process. The 
Poisson process is preferred because it is more accurate for modeling repairable system. 
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Robinson used a nonparametric Bayes approaeh for modeling reliability growth 
(Robinson 1989, 591-598). Robinson’s paper compares using a Bayes analysis technique 
in comparison to the AMSAA PM2-C model (or Poisson model). He states that the Bayes 
approach (or Bayes Theorem) has smaller mean-square prediction errors in comparison to 
Poisson’s. In addition, his research discussed that Bayes typically preforms better than 
AMSAA’s (the Poisson model) or any of nonparametric models used. However, Bayes 
analysis technique is not the chosen reliability model across Department of Defense (DoD) 
in comparison to AMSAA’s (the Poisson model). An exception to the Bayes technique is 
when no information about the failure rates of the components or system is known or 
available. Since ACAT Level I programs likely do not have system failure data available, 
applying Bayes is not applicable. This is two-fold as AMSAA is the standard for use 
amongst the DoD; using a different modeling approach is likely to be less accepted. 
Therefore, it is more practical for ACAT Level I programs to utilize an AMSAA PM2-C 
model, which uses a nonhomogeneous Poisson process to demonstrate and predict 
reliability modeling. I discuss the application of a nonhomogeneous Poisson process in this 
thesis. The design methodology developed in this thesis compares the AMSAA PM2-C 
model through an iterative process. The design methodology utilizes the baselined 
reliability growth-planning curve developed in AMSAA’s PM2-C model and compares it 
to a demonstrated curve. 

There are many related works illustrated in this subsection, each addressing 
reliability in various ways, none of which answers the need that exists for ACAT Level I 
programs and the mandated reliability requirement imposed on them. This thesis addresses 
the need that exists for the mandated reliability requirement placed upon ACAT Level I 
programs as it (1) prepares data gathered from the program’s documents as an input to the 
AMSAA PM2-C model to develop a baselined reliability growth planning curve, (2) 
produces a demonstrated reliability growth curve based on system test data, then (3) 
compares the baselined planning curve to the demonstrated reliability growth curve to 
understand the if the requirement has been met. These tasks are encapsulated within design 
methodology, which is presented as a repeatable process applicable to ACAT Level 1 
programs. 


9 



THIS PAGE INTENTIONALLY LEET BLANK 


10 



III. DATA PREPARATION 


This data preparation chapter addresses what is required to produce the baselined 
curve from the AMSAA PM2-C model. Data preparation is Step 1 of the reliability growth 
planning portion of the methodology (see Figure 1 Methodology Flowchart) provided in 
the introduction. The AMSAA PM2-C model is key to providing the baselined reliability 
growth planning curve that is utilized in the design methodology (discussed in Chapter V). 
This chapter discusses the information needed as an input to the AMSAA PM2-C model 
as well as any transformations or interpretations of that information. Due to the structuring 
of ACAT Level 1 programs and their documentation, the subsections in this chapter are 
organized by the standard program documents from which the information is being 
gathered. That said, the information being gathered is the focus of this chapter. At the end 
of this chapter, a table summarizes all data required as well as the standard documents from 
which it is gathered. 

A, CAPABILITIES DEVELOPMENT / PRODUCTION DOCUMENT 

ACAT Level I programs are required to have a reliability requirement. Inside the 
ACAT Level I program’s reliability requirement(s), the Mean Time Between Failure 
(MTBF) requirement needs to be gathered. MTBF is an average time-to-failure for a 
repairable system, where the time-to-failure is a continuous random variable. In order to 
test to this continuous random variable, the random variable is assumed constant over the 
duration of the test phase and thereby assigning an exponentially increasing probability of 
failure function across time. For example, the program’s reliability requirement will state 
something to the effect of ''the system shall have a Mean Time Between Failure (MTBF) 
requirement greater than or equal to 100 hours." This requirement is either a Key 
Performance Parameter (KPP) or Key System Attribute (KSA) for the system and can be 
found in the Capabilities Development (or Production) Document (CDD or CPD). It is 
important to note that reliability requirements come in a lot of different forms. For example, 
one might say simply there shall be no single point of failure, whereas another may state 
this requirement in reference to the system’s life cycle (i.e., shall be greater than 90% at 10 
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years). The MTBF is a (and necessary) requirement that is always found in the requirement 
documentation for ACAT Level I programs. As ACAT Level I programs can also have 
similar reliability requirements (like mentioned above), it must have an MTBF 
requirement. It is this requirement (MTBF) that is necessary and captured from the 
CDD/CPD during the data preparation step. 

In addition to the MTBF requirement, the level of confidence of the MTBF 
requirement can be found in the CDD/CPD. ACAT Level I programs will most always 
have confidence levels. A confidence level is a probability in which a population parameter 
falls within a specified range of values. A confidence level is the probability the parameter 
is within the confidence interval. The confidence interval is calculated inside the AMSAA 
PM2-C model. Confidence levels are required due to the complexity of such ACAT Level 
I programs, and is the reason behind why such programs reliability requirement is tested 
discretely versus continuously. Although it is true that with continuous reliability testing a 
confidence level is not required, it is with the discrete testing events of an ACAT Level I 
program. Therefore, such programs will have confidence levels for their reliability 
requirement stated in the programs CDD/CPD. This level of confidence percentage is 
inputted into the AMSAA PM2-2 model to depict the Lower Confidence Bound (LCB) of 
the Initial Operational Test (lOT). It is common across the DoD to have a level of 
confidence of 80% (0.80). If the confidence level percent is not obtained inside the 
CDD/CPD, then 0.80 will suffice. 

B, TEST AND EVALUATION MASTER PLAN 

The test schedule can be found in the Test and Evaluation Master Plan (TEMP). 
The importance of capturing the program’s testing schedule is to have the test phase names 
and corresponding hours to be tested for the duration of the acquisition milestone. In 
addition, ACAT Eevel I programs anticipate having downtime between each test phase. 
An ACAT Eevel I program uses this downtime to undergo a series of checks and 
maintenance opportunities prior to beginning the next test phase. This is called a Corrective 
Action Period (CAP). The AMSAA PM2-C model assumes that the user is aware of what 
a CAP is and if it will be used or not. As CAPs may not be utilized in smaller programs, it 
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is normal for ACAT Level I programs to have them factored into the test schedule. This 
reduces the number of test phases to be reran due to the dollar value in executing the test 
phase. The AMSAA PM2-C model calls out this CAP period as “lag time” and asks for its 
number of hours. A “lag time” that is left blank gives the program the flexibility to 
designate the appropriate amount of hours needed should the program need to address (or 
correct) an obstacle. The AMSAA PM2-C model takes into account the number of test 
phases, the hours to be attained during each phase, and predicts a reliability growth curve 
using the Poisson process. This curve acts as a baseline for the program’s reliability growth 
and is the foundation for the iterative design methodology process. 

The AMSAA PM2-C model also requires the planned number of hours for the 
duration of the lOT, which is also found inside the TEMP. Since a single lOT event 
typically does not have enough hours to stand alone, the number of planned hours is 
inclusive to all integrated test events (i.e., the lOT event and all other integrated test hours 
are summarized together). The AMSAA PM2-C model factors in these hours as another 
opportunity for predicting reliability growth. 

The TEMP is also the location for where to find the system’s degradation factor. 
The system’s degradation factor is required for calculating the expected reasonable 
prediction of operational MTBE as the system moves from Developmental Test (DT) to 
lOT. The degradation factor is a number that ranges from 0.00-1.00 that is multiplied to 
the initial MTBE in order to produce a reasonable predictive MTBE for lOT. The AMSAA 
PM2-C model states that the default degradation factor is 0.10. This can be misguided as 
majority of ACAT Eevel I programs conduct a final CAP following DT to fix any failures 
prior to starting lOT. In this case for ACAT Eevel I programs, the initial MTBE is its 
predictive MTBE for lOT, and therefore, a degradation factor of 0.00 is used. The majority 
of ACAT Eevel I programs use 0.00 as the degradation factor as this allows the program 
to be assessed in its true operational environment setting with all of its components failure- 
free. Also, a degradation factor of 0.00 is the most conservative value. The greater the 
degradation value, the greater the planned drop in reliability (MTBE) is associated when 
transitioning from the initial MTBE during DT to the predictive MTBE for lOT. Typically 
degradation factors greater than 0.10 would require subsequent justification as to why the 
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ACAT Level I program has a greater risk to produeing smaller predietive MTBF during 
lOT than its initial MTBF. 

DT planning parameters also reside in the system’s TEMP. These parameters 
inelude (1) the MTBF requirement (this was found earlier in the CDD/CPD), (2) the 
management strategy pereentage, and (3) the system’s average Fix Effeetiveness Eaetor 
(EEE). The management strategy is fraetion in whieh the initial failures of the program are 
planned to be addressed through CAPs in-between DT events. The management strategy 
is portrayed as a pereentage in the AMSAA PM2-C model. It is typical for an ACAT Eevel 
I program to have a management strategy percentage of 0.95. This is aligns with the 
conservative approach taken with the degradation factor of 0.00. The PEP is the percent 
estimate for correcting any failure modes experienced during the test phases. The PEP is 
not the same as the degradation factor. Both the PEP and degradation factors are separate 
values, and both can be found in the TEMP. The AMSAA PM2-C model utilizes these for 
projecting the baselined reliability growth-planning curve. 

C. PERFORMANCE SPECIFICATION 

The Performance Specification (PSPEC) is a document created and intended for the 
Original Equipment Manufacturer (OEM) (i.e., the prime contractor), in order to model 
and procure the system. The Probability of Acceptance (POA) can be found inside the 
PSPEC. The POA is the probability that the government will accept the system that the 
OEM built. The POA assists in calculating the consumer and producer risk probabilities 
inside the AMSAA PM2-C model. Purthermore, it is because of this particular requirement 
(the POA), that the AMSAA PM2-C model is required. 

D, ASSISTANT SECRETARY OF ACQUISITION THRESHOLD 

The AMSAA PM2-C model asks for the Assistant Secretary of the Army 

Acquisition, Eogistics, and Technology (ASA (ALT)) threshold. This threshold is not 

captured in any of the program’s documents. In addition, the AMSAA PM2-C model does 

not clearly specify what this threshold is nor how to calculate it (U.S. Army Materiel 

Systems Analysis Activity 2011, 7). In researching how to generate this value, it was found 

that this threshold requires calculating 70% of the reliability threshold requirement with 
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50% statistical confidence at the end of the first test phase (Dalton 2010, 116). For ACAT 
Level I programs, testing will not have been eonducted and therefore, the value will need 
to be predieted. Further, the information being identified in this ehapter is applied to for 
ACAT Level 1 programs, via the AMSAA PM2-C model, prior to any testing being 
eonducted. Therefore, by default, 0.70 is the pereentage to be demonstrated (the threshold 
requirement for ASA (ALT)). The ASA (ALT) value of 0.70 is the most conservative, and 
in keeping with the approaeh taken with the degradation factor and management strategy 
pereentage, it is the standard approach for ACAT Level I programs to take. As it is truly 
up to the program’s discretion as to the pereent threshold of the reliability requirement the 
program aspires to meet by the end of the first test phase, ACAT Level I programs, typieally 
have values ranging between 0.70-0.80 (i.e., 70-80%) for the ASA (ALT) requirement 
threshold (Dalton 2010, 117). Due to the complexity of ACAT Level I programs, reaehing 
100% of the reliability requirement is typically not seen until the program is rewarded 
entranee into the Full Rate Production (FRP) acquisition milestone. Likewise, 0.50 is 
inputted into the confidence level for ASA (ALT) LCB (the percent statistical confidence 
as required by the threshold). A confidence level of 0.50 is the minimum the AMSAA 
PM2-C model will accept for this value, and it is up to the program’s discretion as to the 
pereent confidence level that the reliability requirement will be met at the end of the first 
test phase. In many eases, ACAT Level I programs will choose to use 0.50 as it is the 
AMSAA PM2-C model’s default value and the outcome of the first test phase is unknown. 

E. RETRIEVED DATA 

Table 1 Data and Document Loeation summarizes the data retrieval process 
executed in this chapter and puts it in a user-friendly format to ensure that all materials are 
ready for produeing the baselined reliability growth-planning curve from the AMSAA 
PM2-C model. 


15 



Table 1. 


Data and Document Location 


Data Required for the 
AMSAA PM2-C model 

Standard Document Location Where 
the Data can he Found 

MTBF requirement 

Capabilities Development/Production 
Document 

MTBF confidence level 

Capabilities Development/Production 
Document 

Test schedule 

Test and Evaluation Master Plan 

Planned hours for lOT 

Test and Evaluation Master Plan 

Degradation factor 

Test and Evaluation Master Plan 

Management strategy 

Test and Evaluation Master Plan 

Average Fix Effectiveness 
factor 

Test and Evaluation Master Plan 

Probability of Acceptance 

Performance Specification 

Assistant Secretary of the 
Army Acquisition, Logistics, 
and Technology (ASA [ALT]) 
threshold 

Quantified 
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IV. PRODUCING THE BASELINED CURVE 


The purpose of this chapter is to clearly and concisely produce the baselined 
reliability growth-planning curve from the AMSAA PM2-C model. This is Step 2 of the 
methodology (see Figure 1 Methodology Flowchart), and is another step in reliability 
growth planning. 

Due to the AMSAA PM2-C model’s manual having significant ambiguity, it is not 
clear as to whether the baselined reliability-planning curve is being correctly produced, 
which is the motivation for including this chapter. In addition, the model can be easily 
misunderstood and therefore misapplied. This case can lead to a scenario where the results 
are incorrect and the program would not be able to identify it as such because the process 
caused the error. Further, when using the AMSAA PM2-C model on an ACAT Level 1 
program, there does not exist a requirement to ensure the process was applied correctly 
(e.g., verified by an AMSAA Subject Matter Expert (SME)), which amplifies the need to 
have clarity in the process to implement the model. One example issue is the belief that 
one should be updating the model following the conclusion of each test phase (U.S. Army 
Materiel Systems Analysis Activity 2011, 4). This error stems from the following text in 
the AMSAA documentation, “enter each test phase as it is used for early planning and 
automatically generate or update totals” (U.S. Army Materiel Systems Analysis Activity 
2011, 4). This is not true because the test phases and corresponding hours are not to be 
automatically generated or updated after being inputted. In order to mitigate any misusage 
of the AMSAA PM2-C model, a walk-through of the model is provided. This ensures that 
the data captured in the data preparation chapter is applied correctly for baselining the 
reliability growth planning curve model for executing the design methodology. 

Baselining the reliability growth-planning curve is an integral step in securing an 
accurate design methodology. If the baselined planning curve is not produced correctly, 
then executing the design methodology may not generate the target outcome, which is to 
demonstrate reliability growth for ACAT Eevel I programs. The AMSAA PM2-C model 
receives the data gathered in Chapter III, and delivers a baselined reliability growth¬ 
planning curve upon executing the model. The model executes this through a Poisson 
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process. A Poisson process is a discrete probability distribution occurring over a fixed time 
where each event occurs constantly and independently of the previous event. 

A. IDENTIFYING THE PURPOSE 

The AMSAA PM2-C model asks to identify the purpose of the DT. The model 
intuitively is asking, “Does the program wish to grow to a level of reliability that will allow 
for demonstration of the requirement with confidence in a follow-on demonstration test 
(e.g., lOT)?” The answer is yes. If not, the AMSAA PM2-C model will not have the ability 
to predict the system’s reliability growth. Also, if no is selected, the reliability growth 
planning curve is a straight line with no growth. 

B. THE SYSTEMS RELIABILITY REQUIREMENT 

The AMSAA PM2-C model asks for the system’s reliability requirement. This is 
the value for the program’s reliability requirement (i.e., MTBF). The program’s reliability 
requirement is input into the “MTBF Requirement (Mr)” designation. If the program has 
multiple reliability requirements, there will need to be a separate baselined reliability 
growth-planning curve depicted for each one. The AMSAA PM2-C model also asks for 
the confidence level at lOT and the program’s POA. 

C. DETERMINING THE lOT PROFILE 

As the lOT profde is not shown, this seetion is included for completeness purposes 
as to understanding all aspects of data inputted into the AMSAA PM2-C model. The 
AMSAA PM2-C model uses the anticipated lOT hours (i.e., the planned hours for lOT) to 
calculate various performance metries. These performance metrics that makeup the lOT 
profile include (1) the maximum allowable number of failures, (2) the goal MTBF 
requirement, (3) the aetual confidence level using a Lower Confidence Bound (LCB), (4) 
the LCB requirement, and (5) the POAs, using the LCB, for both the consumer and 
produeer. This is where the AMSAA PM2-C model determines the lOT profile to depict 
the planning curve’s operating characteristic methodology. 
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D, DETERMINING THE DT GOAL 

“Assume an lOT degradation faetor that reflects a planned drop in reliability” (U.S. 
Materiel Systems Analysis Activity 2011, 5). Although it is not intuitive here, the AMSAA 
PM2-C model asks for the system’s degradation factor. The AMSAA PM2-C model takes 
the degradation factor found in the TEMP to determine the DT Goal. Once inputting the 
anticipated degradation factor in to the AMSAA PM2-C model, it will calculate and 
determine the goal MTBF number expected during DT. 

E, CAPTURING THE DT PLANNING PARAMETERS 

The AMSAA PM2-C model asks for the DT planning parameters, but does not 
specify what they are, or in which order they are inputted, which can be confusing. 
Together the initial reliability requirement (i.e., MTBF), the management strategy, and the 
average FEF make up the program’s DT planning parameters. 

The AMSAA PM2-C model uses the test schedule (i.e., the test phase names and 
individual test hours) for establishing the intermediate goals (opportunities) for reliability 
growth. In addition to the test phase names and hours, the AMSAA PM2-C model also 
asks if “lag time” will be included. 

F. APPLYING THE ASA (ALT) THRESHOLD 

The final constraint applied in the AMSAA PM2-C model is the ASA (ALT) 
threshold. Recall that this threshold is 0.70 with 0.50 statistical confidence for AC AT Level 
1 programs. The AMSAA PM2-C model now is ready to produce the baselined reliability 
growth-planning curve. This curve is utilized in the design methodology chapter. 

G. AMSAA PM2-C BASELINED CURVE 

At this point, all the data is applied into the AMSAA PM2-C model (collected in 
Table 1 Data and Document Location), producing a baselined reliability growth planning 
curve upon clicking “run model.” An example of the model’s baselined planning curve is 
provided in Figure 2 Example of Baselined Planning Curve. 
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Figure 2. Example of Baselined Planning Curve 
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V. DESIGN METHODOLOGY 


The design methodology utilizes the baselined reliability growth planning eurve 
produeed in the previous chapter and compares it to demonstrated reliability growth curve 
developed in this chapter. The design methodology accurately and consistently answers the 
mandated DoDI 5000.02 requirement (USD AT&L 2014, 3-49) for reliability growth for 
ACAT Level I programs. As it states in the DoDI 5000.02, “Reliability Growth Curves 
(RGCs) will reflect the reliability growth strategy and be employed to plan, illustrate, and 
report reliability growth. RGCs are included in the TEMP and submitted at the conclusion 
of Milestone B. RGCs are in a series of intermediate goals and tracked through fully 
integrated, system-level test and evaluation events at least until the reliability threshold is 
achieved. If a single curve is not adequate to describe overall system reliability, curves for 
critical subsystems should also be employed” (USD AT&L 2014, 3). 

Recall this thesis (1) prepares data gathered from the program’s documents as an 
input to the AMSAA PM2-C model to develop a baselined reliability growth planning 
curve, (2) produces a demonstrated reliability growth curve, then (3) compares the 
baselined planning curve to the demonstrated reliability growth curve to understand the if 
the requirement has been met. These tasks are encapsulated within the design methodology, 
which is presented as a repeatable process applicable to ACAT Level 1 programs. The 
sections of Chapter V cover the reliability growth analysis and program execution steps to 
the flowchart (Steps 3-6 respectively) shown in Figure 1 Methodology Flowchart. Recall 
that the data prepared in Chapter III and the production of the baselined planning curve in 
Chapter IV act as (Step I and Step 2 respectively) the reliability growth planning steps to 
the design methodology and therefore and are reminded for completeness. 

A, PRODUCE DEMONSTRATED MTBF 

As each test phase result is acquired, the result is plotted to the baselined reliability 
growth-planning curve. The test phase result (i.e., the demonstrated MTBF) is quantified 
by the test hours for the test phase divided by the number of failures experienced during 
the test phase. For example, an MTBF of 100 hours is at the result of a 500-hour test phase 


21 



experiencing five failures. In addition, in the case where a test phase is executed without 
any failure, the demonstrated MTBF equates to the number of test hours the test phase 
experienced. Be sure to plot this result in alignment to the appropriate test phase related. 

Figure 3 Comparison between Demonstrated (green dot) and Planned (black line) 
MTBF, provides an example of the execution of this step. The test phase’s acquired 
result (i.e., coordinate) is represented by the green dot, which is in alignment to the 
corresponding test phase. 
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Figure 3. Comparison between Demonstrated (green dot) and Planned (black line) MTBF 
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B, COMPARE PLANNED VERSUS DEMONSTRATED MTBF 

In this step of the design methodology, the loeation of the test result to the planning 
curve is assessed. Does the test result lie on or above the baselined reliability growth 
planning curve or, does the test result fall below the planning curve? Step 5 in the 
methodology (see Figure 1 Methodology Flowchart) analyzes these questions. In the 
following sections, are detailed explanations answering these questions. 

1. Positive Reliability Growth 

Recall the initial question in this step of the design methodology was: Does the test 
result lie on the baselined reliability growth-planning curve? Furthermore, does the test 
result fall above the planning curve? If yes, then greater reliability growth was experienced 
than initially anticipated. This is a good thing. It demonstrates that the testing is producing 
positive results and therefore growth from the changes introduced during the CAPs (i.e., 
“lag time”) between test phases. 

The only instance where the outcome of the test would not immediately demonstrate 
reliability growth is at the initial plot. The first test phase result plotted to the baselined 
planning curve only compares its relevance to the planned curve. It is important to report 
on the accomplishments that led to positive reliability growth. The Program Manager (PM) 
uses this report to ensure such accomplishments and recommendations take precedence in 
the next test phase. An example of a reliability accomplishment report is provided in Table 
2 Reliability Accomplishment Report. 
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Table 2. 


Reliability Accomplishment Report 


Issue 

Accomplishment? 

Why? 

Recommendation 

New 

synchronizers 
placed in 
parallel allow 
failure to 
occur during 
Test phase 
Bravo 

Yes 

An engineering 
enhancement 
allowed the 
synchronizers to 
fail without 
interfering with 
the mission. 

Continue to monitor the new 
synchronizers placed in parallel 
to ensure that they experience 
true parallel failure over the 
lifespan and not increased 
failure rate after the initial 
failure. 


2, Negative Reliability Growth 

Recall the initial question in this step of the design methodology was: Does the test 
result he on the baselined reliability growth-planning curve? Furthermore, does the test 
result fall below the planning curve? If the test result falls below the planning curve, then 
less reliability growth was experienced than what was anticipated. This is not a good thing. 
Unfortunately, not ah test phases will result in positive reliability growth. It is normal to 
expect that at least one test phase will not achieve its anticipated planned reliability growth. 

The identification that the demonstrated MTBF is lower than expected can be used to 
shift efforts to correct the obstacles experienced. Shifting efforts, such as incorporating 
design changes during CAPs to mitigate this obstacle from occurring in future test phases. 
This offers a way to mitigate the obstacles experienced from occurring in future test phases. 
It is important to report on the obstacles that led to less than desired (negative) reliability 
growth. An example of a reliability obstacle report is provided in Table 3 Reliability 
Obstacle Report. 
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Table 3. 


Reliability Obstaele Report 


Issue 

Obstacle? 

Why? 

Recommendation 

New run 
flat tire 
deflated at 
a fast rate 
during Test 
event Alpha 

Yes 

This was due to the 
inability to continue 
maneuvering at the 
request of the 
mission. 

Have spare run flat tires on-hand 
so the mission can still be 
executable should the incident 
occur again. 

Root-cause to determine if this is 
an infant mortality (isolated issue) 
or if there is a manufacturer’s 
defect in the deflation of the run 
flat tire. 


3, Repeat Previous Steps throughout All Test Phases 

As eaeh test phase is exeeuted, it is plotted on the baselined planning curve until all 
test phases are completed. This is displayed in Figure 1 Methodology Flowchart as the 
question “testing completed.” It is at this location in the flowchart that if not all the testing 
phases are completed Steps 3-5 are executed again, until the completion of all the test 
phases. In addition, if the result of the test phase produced positive or negative reliability 
growth, an appropriate reliability accomplishment/obstacle report will accompany. All of 
the reliability accomplishment/obstacle reports are compiled into a single report at the end, 
but it is important to have this done iteratively, as it is easy to recall. Figure 4 Plotted Test 
Phase Results to Planned Curve shows an example of what the curve will look like with all 
the test results plotted to the baselined planning curve. It is typical for ACAT Level I 
programs to have a tight testing schedule and are highly encouraged to execute to this 
schedule to keep the program aligned. Therefore, it is highly unlikely that ACAT Level I 
programs will go back to a previous test due to any failures found. In this case, the failures 
(i.e., obstacles) are noted and mitigated going into the next test phase. There is no room in 
programs of the ACAT Level I designation for repeating test phases. 


26 






Test Time 


Figure 4. Plotted Test Phase Results to Planned Curve 
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4, Develop a Trend Line 

After plotting all the test phase results to the baselined planning eurve, trend 
analysis is eondueted on the aehieved results. This trend analysis is performed either 
linearly or exponentially, and is important as it will beeome the pietorial that is briefed to 
the MDA for reward into the next aequisition milestone of the program. 

The purpose for applying a trend line is to develop a best-fit line from the data sets 
(i.e., demonstrated MTBF). The trend line is applied to the test phase results that are plotted 
to the baselined planning eurve. A linear trend line is used when the data sets 
inerease/deerease at a steady rate. Otherwise, an exponential trend line is used when data 
sets inerease at a higher rate. Reliability growth oceurs when an ACAT Level I program is 
able to suecessfully identify failures and improve upon the design of the system to reduce 
the rate at which failures surface in future test phases. Figure 5 Linear Achieved Trend 
Plotted to Planned Curve provides an example of a linear trend on the achieved test results 
plotted to the baselined planning curve. This trend line acts as the program’s demonstrated 
curve, which is compared to the baselined planning curve. This pictorial provides a single 
snapshot of what transpired over the milestone. 
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Figure 5. Linear Achieved Trend Plotted to Planned Curve 
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C. REPORT RESULTS 

The final step of the design methodology is to report the program’s overall 
reliability growth aehieved. This is the program execution portion of the methodology (see 
Figure 1 Methodology Flowchart). ACAT Level I programs are required to track, analyze, 
and report their reliability measurements on a regular basis to the MDA. The program 
provides the MDA with a single reliability report and achieved trend to the baselined 
planning curve in order to provide the proper justification for reward into the next 
acquisition milestone. Although this step is much more formalized in a thorough reliability 
growth report for ACAT Level I programs, this thesis captures the process in which to meet 
the mandated reliability growth requirement imposed on such programs as opposed to 
discussing what a formalized reliability report should look like. 

It is critical that this report include the baselined planning curve compared to its 
demonstrated curve in order to show to the MDA that reliability growth has occurred (i.e., 
results). In addition, the reliability growth report should include the reliability 
accomplishment/obstacle report (i.e., recommendations) in support of the program’s 
reliability growth achievement. 

D, LIMITATIONS 

Limitations to this design methodology include the difficulty to factor in reliability, 
should the testing schedule not incorporate CAPs in between phases in order to test-fix- 
test. As the AMSAA PM2-C model does develop a curve based off the exclusion of CAPs, 
it is more difficult, and this thesis did not discuss that particular instance. 

Another limitation to this design methodology is the case where the program’s 
reliability requirements are changed in the middle of an acquisition milestone. It is assumed 
that the change can be noted and the future test phases begin testing to this new 
requirement. 

A similar drawback to this limitation is if the program is forced to be fielded, in 
order to fill an urgent capabilities gap in the fleet, prior to the program determining if it 
meets its reliability requirement. Likewise, it is assumed this too can be noted, and the 
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fielded prototype be monitored and assessed with its true operational data once it is made 
available. That said, this work did not investigate the changes mentioned here. 

Lastly, it should be noted that by executing this design methodology, it does not 
mean that the ACAT Level I program will meet its reliability requirement. This design 
methodology acts only as a means to produce a baselined reliability growth planning curve 
that test results can be plotted to, in order to track the progress and ensure that the program 
is trending in the correct direction. 

In addition, any errors in the design methodology are mitigated by ensuring that the 
baselined planning curve is produced accurately in the previous chapter. The only cases 
where executing the design methodology would not be applicable is when the legacy 
program that the ACAT Level I program is replacing goes obsolete early and the ACAT 
Level I program is to be fielded in order to close the capability gap that exists in the fleet 
immediately. During such instance, the design methodology may not be required as the 
tested prototype is required for operational use and reliability growth concerns are placed 
on hold and assessed at a later time. 

E. SUMMARY 

The design methodology uses a comparative process where it takes the baselined 
planning curve from the AMSAA PM2-C model and compares it to the demonstrated curve 
(from the acquired test phase results). This is done to assess the predicted reliability growth 
to the demonstrated reliability growth. 

The design methodology utilizes the baselined reliability growth-planning curve 
produced by the AMSAA PM2-C model and compares it to a demonstrated reliability 
growth curve. Recall that these complex programs must grow reliability into their schedule 
and test throughout their current acquisition milestone in order to allow changes to occur 
intermittently between, or at the completion of the events, to produce overall reliability 
growth and grant entrance into the next milestone. As a result, the design methodology 
allows ACAT Level I programs to assess their system’s reliability requirement by 
comparing its planned reliability curve to its demonstrated curve. 
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VI. CASE STUDY DEMONSTRATION 


The case study chapter implements the data preparation, producing the baselined 
curve, and design methodology chapters. This case study looks at the Phoenix Talon AC AT 
Level I Unmanned Aerial Vehicle (UAV) program. Phoenix Talon has just completed (on 
schedule) its Milestone B test phases. Phoenix Talon is compiling its reliability growth 
report to give to the MDA for entrance into Milestone C. 

A, PHOENIX TALON DATA PREPARATION 

The data for the Phoenix Talon was prepared in the following table (see Table 4 
Phoenix Talon Data Preparation). This data was used in producing the baselined reliability 
growth planning curve from the AMSAA PM2-C model. 


Table 4. Phoenix Talon Data Preparation 


Step 1: Identify the Purpose of your Developmental Test (DT) 

Do you wish to grow to a level of reliability that will allow you to demonstrate the 
requirement with confidence in a follow-on demonstration test? 

Yes 

Step 2: Input the Program Requirements 

MTBF Requirement (MR) 

150 

Confidence Level for lOT LCB 

0.80 

Prob. Of Acceptance in lOT using LCB 

0.50 

Step 3: Determine the lOT Profile Using OC Curve Methodology 

lOT Test Duration 

3,500 

Step 4: Determine the DT Goal 

Assumed DT to lOT Degradation Factor 

0.00 


Table cont’d on next page 
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Table cont’d from previous page 


Step 5: Input the DT Planning Parameters 

Initial MTBF (MR) 

85 

Management Strategy (MS) 

0.95 

Average FEF 

0.70 

Step 6: Input the DT Schedule 

Test 

Phase 

Name 

DTI Alpha 

Mission Time 

800 

CAP at End of Phase? 

Yes 

Test 

Phase 

Name 

DT2 Alpha 

Mission Time 

500 

CAP at End of Phase? 

Yes 

Test 

Phase 

Name 

DTI Bravo 

Mission Time 

750 

CAP at End of Phase? 

Yes 

Test 

Phase 

Name 

DT2 Bravo 

Mission Time 

600 

CAP at End of Phase? 

Yes 

Test 

Phase 

Name 

DTI Charlie 

Mission Time 

1,000 

CAP at End of Phase? 

Yes 

Test 

Phase 

Name 

DT2 Charlie 

Mission Time 

500 

CAP at End of Phase? 

Yes 

Test 

Phase 

Name 

DTI Delta 

Mission Time 

500 

CAP at End of Phase? 

Yes 

Test 

Phase 

Name 

DT2 Delta 

Mission Time 

850 

CAP at End of Phase? 

Yes 


Table cont’d on next page 
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Table cont’d from previous page 


Test 

Phase 

Name 

DTI Echo 

Mission Time 

1,500 

CAP at End of Phase? 

Yes 

Test 

Phase 

Name 

DT2 Echo 

Mission Time 

600 

CAP at End of Phase? 

Yes 

Step 7: Apply the ASA(ALT) Threshold 

Test Phase for ASA(ALT) Threshold 

N/A 

Percent of MR to be Demonstrated 

0.70 

Confidence Level for ASA(ALT) LCB 

0.50 


B, PHOENIX TALON PRODUCES BASELINED CURVE 

After retrieving all the data needed for the AMSAA PM2-C model, the Phoenix 
Talon produced the following curve (see Figure 6 Phoenix Talon’s Baselined Curve). 
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Phoenix Talon - PM2-Continuous Reliability Growth Planning Curve 
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Figure 6. Phoenix Talon’s Baselined Curve 
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C. PHOENIX TALON EXECUTES DESIGN METHODOLOGY 

The Phoenix Talon successfully completed their first test phase called “DTI 
Alpha.” This test phase produced an MTBF of 100 hours. This acquired test phase result 
was plotted on the Phoenix Talon’s baselined curve, and is displayed in Figure 7 Phoenix 
Talon Plotted Test Result to Planned Curve. 
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Figure 7. Phoenix Talon Plotted Test Result to Planned Curve 


38 






























In following the design methodology, the plotted test result was analyzed to assess 
whether reliability growth oeeurred. Figure 7 Phoenix Talon Plotted Test Result to Planned 
Curve shows that the 100-hour MTBF falls on the planning curve line. An accompanying 
reliability obstacle report was provided for DTI Alpha. This report can be found in Table 
5 Phoenix Talon Reliability Obstacle Report for DTI Alpha. It was determined that the 
eight failures experienced throughout DTI Alpha were due to a misalignment in the 
system’s software. In particular, the Phoenix Talon’s software had timing issues residing 
inside its communications to its electronic automatic timing synchronizer. 


Table 5. Phoenix Talon Reliability Obstacle Report for DTI Alpha 


Issue 

Obstacle? 

Why? 

Recommendation 

Electronic 

automatic 

timing 

synchronizer 

misalignment 

with 

software 

Yes 

This was due to software 
programming crashes causing 
the Phoenix Talon to become 
unresponsive during parts of 
the mission. 

During next CAP 
opportunity, provide 
software updates in 
order to mitigate the 
issue in follow-on 

tests. 


The Phoenix Talon continued to execute its DT schedule with various CAPs 
determined in-between to address and correct reliability issues when they arose. As each 
test phase concluded, the phase’s reliability result was acquired. The test phase’s reliability 
results are displayed in Table 6 Phoenix Talon’s Test Results. 
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Table 6. 


Phoenix Talon’s Test Results 


Test Phase 

MTBF 

DTI Alpha 

100 hours 

DT2 Alpha 

100 hours 

DTI Bravo 

125 hours 

DT2 Bravo 

150 hours 

DTI Charlie 

200 hours 

DT2 Charlie 

166.7 hours 

DTI Delta 

250 hours 

DT2 Delta 

170 hours 

DTI Echo 

187.5 hours 

DT2 Echo 

200 hours 


The Phoenix Talon’s test results were plotted to the baselined eurve. In addition, 
the test results were used to perform a trend analysis. This linear trend analysis was plotted 
to the planned eurve. Figures of both steps are provided Figure 8 Phoenix Talon’s Plotted 
Test Phase Results to Planned Curve and Figure 9 Phoenix Talon’s Linear Aehieved Curve 
Plotted to Planned Curve respectively. 
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Figure 8. Phoenix Talon’s Plotted Test Phase Results to Planned Curve 
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Figure 9. Phoenix Talon’s Linear Achieved Curve Plotted to Planned Curve 
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Following the alignment of the design methodology chapter, following the linear 
trend plot to the planned curve, the Phoenix Talon’s reliability accomplishment/obstacle 
report was produced, and is provided in Table 7 Phoenix Talon’s Reliability 
Accomplishment/Obstacle Report. 
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Table 7. 


Phoenix Talon’s Reliability Aeeomplishment / Obstaele Report 


Issue 

Accomplishment/ 

Obstacle? 

Why? 

Recommendation 

Receiver/Exciter 
Transmitter 
misalignment 
with software 

Obstacle 

This became an obstacle periodically 
during DTI Alpha due to software 
programming crashes causing the 
Phoenix Talon to become 
unresponsive during parts of the 
mission. 

During next CAP opportunity, 
provide software updates in 
order to mitigate the issue in 
follow-on tests. 

Receiver/Exciter 
Transmitter 
misalignment 
with software 

Obstacle 

This obstacle remained in DT2 Alpha 
due to funding constraints and not 
being able to get the update 
incorporated prior to executing DT2 
Alpha. 

During next CAP opportunity, 
provide software updates in 
order to mitigate the issue in 
follow-on tests. 

Receiver/Exciter 
Transmitter 
misalignment 
with software 

Accomplishment 

Contractor was able to update the 
software during the CAP prior to DTI 
Bravo to fix the issue. In return, there 
was a greater increase in Mean Time 
Between Failures linked to the failure 
mode. 

At the conclusion of Milestone 
B, if the failure is no longer 
experienced, can close on 
Watchlist and Assumed Risks 
reports. 

Cabling within 
Phoenix Talon to 
he replaced 

Accomplishment 

Testers noticed the cabling becoming 
frayed and requiring replacement after 
mission executions, believed it to be 

To replace the cabling within 
Phoenix Talon once a more 
durable Form, Fit, Function 
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Issue 

Accomplishment/ 

Obstacle? 

Why? 

Recommendation 



due to inereased tension and demand 
than originally antieipated. 

viable replaeement is 
determined. 

Replacement purchased and 
installed during DT2 Bravo. 

Longer mission 
hours surpass 
battery life 
expectancy 

Accomplishment 

Analyst and CDT tested the enduranee 
and longevity of the Phoenix Talon by 
exeeuting longer missions to test the 
power eharging/hold of the battery life 
in DTI Charlie by nearly doubling the 
mission hours for the test event. 

Battery life expeetaney 
surpassed its guaranteed 
amount, which led to inereased 
reliability and easier 
maintenanee of the UAV. 

Image Formation 
Processor 
overheats in 
changed testing 
environment 

Obstacle 

Testers notiee the amount of 
overheating failures related to the 

Image Formation Proeessor during 

DT2 Charlie after reloeating their test 
to the desert. 

Replaee the Image Formation 
Proeessor with a Form, Fit, 
Funetion eomponent that has 
the same speeifieations but ean 
withstand a greater 
temperature range/variation. 

Image Formation 
Processor v2 was 
installed 

Aceomplishment 

Following the CAP, prior to starting 

DTI Delta, the Image Formation 
Proeessor v2 was installed. DTI Delta 
remained testing in the desert 
environment to ensure replaeement 
will handle high temperatures. 

At the conelusion of Milestone 
B, if the failure is no longer 
experieneed, ean elose on 
Watchlist and Assumed Risks 
reports. 
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Issue 

Accomplishment/ 

Obstacle? 

Why? 

Recommendation 

Information 
Architecture (lA) 
patch updates to 
software inhibit 
previously 
working 
capability. 

Obstacle 

Required software update to 
government harden the software to 
deerease the amount of adversary 
vulnerabilities. Due to these patch 
updates; software played a key role in 
the reduction of MTBF results for 
Phoenix Talon. 

Update the software version to 
work more easily with lA 
patches installed. 

Updated 
software version 
works with lA 
patches installed 

Accomplishment 

Software version was updated in the 
Phoenix Talon prior to starting DT2 
Echo. 

At the conelusion of Milestone 
B, if the failure is no longer 
experienced, ean close on 
Watchlist and Assumed Risks 
reports. 
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The final step of the design methodology for the Phoenix Talon is to report the 
program’s overall reliability growth aehieved. The Phoenix Talon was suoeessfully able to 
track, analyze, and report its reliability measurements on a regular basis. 

The Phoenix Talon case study executes the data preparation, production of the 
baselined curve, and design methodology from this thesis to accurately and consistently 
answer the mandated DoDI 5000.02 requirement (USD AT&L 2014, 3-49) for reliability 
growth. As seen in the case study, the design methodology utilized the baselined reliability 
growth planning curve produced by the AMSAA PM2-C model and expanded upon it in 
an iterative manner. As a result, the Phoenix Talon’s reliability growth report and 
accompanying pictorial was provided to the MDA, which led to a successful entrance into 
Milestone C. 
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VII. CONCLUSIONS 


This chapter discusses the significanee (or viability) of the thesis by explaining its 
relevance to the need that eurrently exists. In addition, the eonelusion also identifies future 
researeh, followed by a brief summary of the research question (and therefore needs 
statement) and ease study results. 

A. SIGNIFICANCE OF PAPER 

The signifieanee of the thesis is to artieulate a eoneept that ineorporates data 
preparation, produetion of a baselined planning eurve, and a design methodology to answer 
the need that exists in illustrating reliability growth at the ACAT Level I designation. 

This thesis elearly answers the need of illustrating the reliability growth 
requirement for ACAT Level I programs imposed by the MDA. This is aecomplished 
through the data gathering, produetion of a baselined reliability growth planning curve, and 
exeeution of the design methodology in a systematie manner. As there have been similar 
methodologies developed, none addressed in the related literature model reliability growth 
in a way that meets the mandated requirement. This is the first instanee where reliability 
growth modeling for ACAT Level I programs elearly and eoncisely artieulate to the MDA 
their reliability aspirations. Exeeuting this eoneept (to include data preparation, production 
of a baselined curve, and design methodology) gives confidence to the MDA that the 
ACAT Level I program will ensure reliability growth suceessfully. 

B, FUTURE RESEARCH 

This subsection highlights the possibilities to expanding upon this thesis for future 
work. In developing the design methodology, it oceurred that the exeeution of these steps 
ean beeome very tedious. In looking into future ways to expand this thesis, it would be 
ideal if there were a single model that ereated a baselined reliability growth-planning eurve 
as well as plot the test phase results as they are aequired. A single environment to iterate 
the baselined curve and design methodology would be nice. In addition, the design 
methodology ehapter also does not expand upon if a system’s reliability growth ean lead 
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to the elimination or shortening of future test phases. As this could be a possibility, the 
design methodology only considered the testing schedule that was part of the data 
collection and preparation in Chapter III. Future work opportunities can include 
researching ways to mitigate test phases should reliability growth be met early on in a 
program’s testing. 

C. SUMMARY 

The MDA has a special interest in ACAT Level I programs and imposes additional 
mandated requirements to ensure the federal governments dollars are spent wisely. One of 
the MDA’s mandated requirements is to ensure that the program has incorporated 
reliability growth modeling. Therefore, the need existed to develop a design methodology 
to fulfill the requirement imposed by the MDA. Should a program of ACAT Level I 
designation not meet these mandated requirements to the MDA, they will not be rewarded 
to move to the next milestone in the acquisition process. The mandated requirement 
reduces the risk of highly expensive programs (i.e., ACAT Level I) testing and delivering 
of unreliable systems to the fleet that do not meet the warfighters need. 

The research methodology answers the question: how do ACAT Level I programs 
demonstrate reliability growth modeling in order to meet the mandated requirement 
imposed by the MDA? This thesis clearly articulates a design methodology that, once 
utilized, can assist ACAT Level I programs in meeting this mandated requirement. As the 
design methodology requires data gathering and the production of a baselined reliability 
growth-planning curve to execute, it is successful as seen in the Phoenix Talon ACAT 
Level I case study. 

Now that this design methodology has been tested through the Phoenix Talon case 
study, we can assume this will assist in other future ACAT Level I programs in modeling 
reliability growth. As a result, the Phoenix Talon was successfully able to track, analyze, 
and report its reliability measurements on a regular basis. In summary, in executing the 
design methodology in this thesis, Phoenix Talon’s reliability growth report and 
accompanying pictorial were provided to the MDA, which led to a successful entrance into 
Milestone C. 
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